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Alliance Helps States Map New Terrain in 
Educator Evaluation 


About five years ago, states across the country took on the huge, complex task 
of developing and implementing new systems to evaluate teacher and principal 
performance in public schools. REL West 

Research Digest keeps 


In response to a federal mandate aimed at 
improving student achievement, especially 
in the lowest performing schools, state 
boards of education drafted high-level 
educator evaluation frameworks that 
called for the use of multiple evaluation 
measures, including growth in student 
achievement. But models of effective 
systems were scarce, and state and local 
education agencies were looking for ideas 
and guidance. 

“These evaluation requirements were part 
of a major shift in federal policy,” says Reino 
Makkonen, a senior policy analyst at WestEd. 
“While previous policy emphasized highly 
qualified educators, measured largely by 
certifications and years of experience, the 
new policy called for evaluating teacher and 
principal effectiveness , as evidenced by job 
performance and student achievement.” 

Makkonen leads the REL West team that 
has provided data analysis and technical 
assistance services to the West Region 
states of Arizona, Nevada, and Utah since 
2012, through the Educator Effectiveness 
Alliance. Members comprise state specialists 
in leadership, instruction, evaluation, and 
assessment. 


“The alliance is looking at evaluation efforts 
through a couple of lenses,” Makkonen 
says. “First, by regulation, states must 
be implementing a set of valid, reliable 
measures for evaluating educators, and 
we’re helping out with technical aspects of 
developing those measures. 

“Second, on the research side, we’re 
supporting states to regularly think through 
and study how well the implementation 
structures and procedures are working,” 
he adds. 


educators and others 
abreast of the research, 
data analysis, and technical 
assistance carried out by the 
Regional Educational Labora- 
tory West ( REL West), includ- 
ing joint efforts with eight 
state and regional research 
alliances of education stake- 
holders in the West Region 


For example, during the 2012/13 and 
2013/14 school years, at the request of the 
Arizona and Utah state education agencies 
(SEAs), REL West supported pilot testing 
of the states’ new standards-based teacher 
evaluation instruments and procedures in 
selected districts. 

As testing progressed, state agencies 
collected data from a variety of sources, 
including the initial results from principals’ 
observations of teachers’ instructional 
practice, parent surveys, and assessments 
of student academic progress, as well 
as feedback on implementation from 
participating teachers and principals via 
focus groups and surveys. 
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states of Arizona, California, 
Nevada, and Utah. In addition 
to providing abstracts of 
selected REL West research 
reports, the twice-yearly 
digest may include 
descriptions of upcoming 
REL West work, services, 
and events in the West 
Region, or publications 
developed by other 
researchers. 
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Visit the REL West website featuring: 


* Collaborative work with 
Research Alliances 

* Free reports on state issues 

* Events that connect research, policy, 
and practice 

http://relwest.wested.org 


Analysis of that data revealed a range of issues, from 
practical needs such as ensuring parent access to 
online evaluation surveys, to concerns that teacher 
observation scores were tending to bunch up at the 
high end of performance scales. 

As REL West begins its third year working with the 
alliance, Makkonen says the group has made steady 
progress in establishing a continuous Plan-Do-Study- 
Act CPDSA) 1 cycle of data collection and analysis 
to inform improvement of evaluation instruments 
and systems. The cycle begins in early summer, 
when researchers begin analyzing data from the 
previous year’s pilot testing. A few weeks later, at the 
beginning of the new school year, they meet with 
alliance members to review their findings, discuss and 
plan needed changes, and help develop new research 
questions for the coming year. 

Although the idea behind the PDSA cycle is 
straightforward, the approach is a whole new way 
of doing business for most state and local education 
agencies, according to Marie Mancuso, associate 
director of the West Comprehensive Center at WestEd. 
“Setting up educator evaluation and support systems 
requires substantial bolstering of the technical 
and systems capacity of most states and districts,” 
says Mancuso. “But developing the needed human 
capacity to put effective educator evaluation systems 
into practice may be the most challenging— and 
important— effort.” 

As associate director of what was then the 
Southwest Comprehensive Center, Mancuso was part 
of collaborative efforts with REL West in 2010 and 
201 I to provide regional educators, researchers, and 
policymakers a basic grounding in the knowledge 
base, models, and emerging best practices for 
educator evaluation. A series of events, convened 
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across states, offered state and district teams an 
opportunity to learn from experts and one another, 
and to start planning first steps toward developing 
teacher and principal evaluation measures and 
systems that would be valid, reliable, feasible, 
and fair. 

Mancuso says she is optimistic about the shifts in 
educator roles and responsibilities she’s witnessed 
over the past four years and its potential to inform 
practice. “As principals receive training and experience 
in conducting structured observations of teachers 
and providing feedback, we’re noting that it’s 
building their capacity as instructional leaders,” says 
Mancuso. “And the development of more objective 
classroom observation tools is prompting teachers 
to ask principals to observe their teaching more 
often than the traditional twice a year— to get a more 
representative view of their skills. 

“It’s shifting the conversations between principals and 
teachers, and teachers and teachers,” she adds, “which 
is an indication that we’re moving toward the bigger 
goal of using data to improve instruction and student 
achievement.” 

Makkonen echoes that observation. “Many principals 
have tended to view their role as managerial,” he 
says. “But if, through professional development and 
support, more school leaders can expand their role 
to include instructional leadership and coaching, that 
kind of fundamental shift could have a direct impact 
on learning.” 

In 2014/15 the focus of the Educator Effectiveness 
Alliance research is also shifting toward these larger 
aims. “Though not perfect, educator evaluation 
measures have improved and the systems used to 
implement them have improved,” says Makkonen. “The 
state agencies now want to look at how districts are 
using these tools to improve learning. 

“At some point in the future we hope these findings 
may also inform evaluation policy.” 

For further information, contact Reino Makkonen at 
rmakkon@WestEd.org or 4 15.6 15.3356. 

Endnote 

1 Concept developed by Walter Shewhart ( 1891-1967), 
an American physicist, mathematician, and statistician. 
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Educator Effectiveness Publications from REL West and other RELs 


R EL West 

Principal and Teacher Perceptions of Implementation 
of Multiple-Measure Teacher Evaluation Systems 
in Arizona 

This study found that teachers and principals in 
districts that piloted new teacher evaluation models 
view standards-based teacher observations as the 
most credible form of evaluation but are open to 
incorporating student performance and stakeholder 
survey measures. (November 2014) 

For further information, contact Reino Makkonen at 
rmakkon@WestEd.org or 415.615.3356. 

Properties of the Multiple Measures in Arizona's 
Teacher Evaluation Model 

This study of Arizona’s pilot teacher evaluation model 
explored the relationships between component 
measures (teacher observations, student academic 
progress, and stakeholder surveys) and investigated 
how well the model differentiated between highl- 
and low-performing teachers. Key findings indicated 
that most teachers were rated “proficient” on most 
observation items, that observation scores correlated 
with student academic progress primarily in domains 
outside the classroom, and that the strength of 
correlation between observation items and student 
academic progress differed for higher and lower 
scoring teachers. (November 2014) 

For further information, contact Reino Makkonen at 
rmakkon@WestEd.org or 415.615.3356. 

o follow us on Twitter @REL_West 


REL Mid-Atlantic 

Using Alternative Student Growth Measures 
for Evaluating Teacher Performance: What the 
Literature Says 

This report describes two alternatives for 
incorporating measures of student achievement 
growth in teacher evaluations: applying statistical 
value-added methods to outcomes other than student 
standardized test scores and using student learning 
objectives. (September 2013) 

Alternative Student Growth Measures for Teacher 
Evaluation: Profiles of Early-Adopting Districts 
This report features the comments of administrators 
in eight districts that were early adopters of 
alternative measures of student growth about 
how they used value-added models and student 
learning objectives to evaluate teacher performance. 
(April 2014) 

Professional Practice, Student Surveys, and Value- 
Added: Multiple Measures of Teacher Effectiveness in 
the Pittsburgh Public Schools 
This study found that each of three measures of 
educator effectiveness— professional practice 
measures derived from the Danielson Framework for 
Teaching; Tripod student survey measures; and value- 
added measures— has the potential to differentiate 
the performance levels of different teachers. The 
three types of measures are also positively but 
modestly correlated with each other. (July 2014) 

REL Northeast & Islands 
How States Use Student Learning Objectives 
in Teacher Evaluation Systems: A Review of 
State Websites 

This report provides an overview of how states define 
and apply student learning objectives in teacher 
evaluation systems. (March 2014) 
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Promising Practices from Alliance Work Featured in 
New REL West Publications 

Some useful findings and promising practices are emerging from REL West’s work with state and regional alliances 
of educators and others committed to data-based decision making to improve teaching and learning. The toolkit 
and reports described below were published in the summer and early fall of 2014. 


Speak Out, Listen Up! Tools for Using Student 
Perspectives and Local Data for School Improvement 

Listening closely to what students say about their 
school experiences can be beneficial to educators for 
understanding and addressing school-related topics 
and problems and rethinking policies and practices. 

The purpose of this toolkit is to provide educators with 
a purposeful and systematic way to elicit and listen to 
student voice to inform school improvement efforts. 

It was developed through the collaborative efforts 
of the Washoe County School District (Nevada) and 
REL West. (July 2014) 

For further information, contact BethAnn Berliner at 
bberliner@WestEd.org or 510.302.4029. 

The Appropriateness of a California Student and Staff 
Survey for Measuring Middle School Climate 

This study examined the appropriateness of 
using student and staff self-report surveys— the 
California School Climate, Health, and Learning 
Survey (Cal-SCHLS)— to assess school climate 
in middle schools. The results suggest that the 


Cal-SCHLS student survey can be used to validly and 
reliably assess climate at school and teacher levels. 
(September 2014) 

For further information, contact: Tom Hanson at 
thanson@WestEd.org, or 562.799.5170. 

Using Assessment Data to Guide Math Course 
Placement of California Middle School Students 

This report offers findings from a study of algebra I 
placements based on different test scores for middle 
school students in California. The study found that 
placement into grade 8 algebra I based solely on 
students’ grade 6 California Standards Test proficiency 
status results in some students taking the course who 
have less than a 40 percent chance of success. The 
study also finds that a newly available assessment 
developed by the Math Diagnostic Testing Project 
makes a valuable contribution to decisions about 
algebra I placement. (September 2014) 

For further information, contact Neal Finkelstein at 
nfinkel@WestEd.org, or 415.615. 3171. 


ASK REL WEST 

Ask REL West is a reference desk service that functions much like a technical reference library. 

Submit your education-related question and REL West will respond with information from IES research 
projects, publications, and reports; references for and bibliographies of other existing education 
research; referrals to federally funded education organizations and websites; and regionally specific 
educational information. To ASK REL West, go to http://relwest.wested.org/contact 


CONTACT INFORMATION 

REL West atWestEd 
730 Harrison Street 
San Francisco, CA 94107 
866.853.1831 
relwest@WestEd.org 
relwest.WestEd.org 
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